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OVERVIEW OF THE NATIONAL BOARD FOR EDUCATION SCIENCES 

Background, 

On November 5, 2002, Congress passed the Edueation Seienees Reform Aet of 2002 
(ESRA), establishing the Institute of Edueation Seienees (lES, or the Institute) and its 
board of direetors, the National Board for Edueation Seienees (NBES, or the Board). The 
Institute reports to Congress yearly on the eondition of edueation in the United States. 
The Institute provides thorough and objeetive evaluations of federal programs, sponsors 
researeh relevant and useful to edueators and others (sueh as polieymakers), and serves 
as a trusted souree of unbiased information on what works in edueation. 

NBES eonsists of appointed and ex offieio members who are highly qualified to appraise 
edueation researeh, statisties, and evaluations (see appendix A). The 15 appointed 
members of the Board are nominated by the President and eonfirmed by the Senate. 
Members Beth Ann Bryan, Caro lin e Hoxby, Gerald Qerry) Lee, and Joseph Torgesen 
eompleted their respeetive terms in November 2008. Board ehair Robert C. Granger 
finished his term in January 2009. NBES eurrently serves under the leadership of the 
eleeted ehair, Erie A. Hanushek, and viee ehair, Jonathan Baron. Currently, the Board is 
operating with nine member vaeaneies. During this reporting period, the Board met in 
September 2008 and January 2009. A meeting seheduled for May 2009 was eaneeled 
due to laek of a quorum. 

From November 2002 to November 2008, Grover J. (Russ) Whitehurst served as the first 
lES direetor. His leadership helped to transform how edueation researeh is planned 
and funded in the U.S. Department of Edueation. In November 2008, Dr. Whitehurst 
published his third and final biennial report to Congress, Rigor and Relevance Redux. 
The report ineluded a detailed aeeount of the aetivities of lES and its four National 
Edueation Centers, as weU as a summary of aU lES grants and eontraets in exeess of 
$100,000 during the biennium. (The full report is available at http://ies.ed.gov/direetor/ 
pdf/20096010.pdf) 

On June 1, 2009, lES weleomed John Q. Easton as Direetor of lES. Dr. Easton was 
appointed by President Baraek Obama to a 6-year term. The Board looks forward to 
working with Dr Easton to aeeomplish the vision and goals of the Institute. 
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MAJOR INSTITUTE OF EDUCATION SCIENCES UPDATES 

American Recovery 
and Reinvestment 
Act ot 2009 ^ARRA^ 

The addition of $100 billion in federal funding for edueation, and a mandate to save 
or ereate jobs and reform sehools, has made 2009 an unpreeedented year for the U.S. 
Department of Edueation. As Seeretary of Edueation Arne Dunean has said, “This is a 
onee-in-a-Hfetime opportunity to do something speeial, to drive ehange, to make our 
sehools better. I want all of us to work hard enough and smart enough to take full 
advantage of this, beeause it’ll never happen again.” 

ARRA has provided the Institute of Edueation Seienees (lES, or the Institute) with 
an additional $245 million in grant funds for state edueation ageneies to further the 
development and implementation of statewide longitudinal data systems through the 
National Center for Edueation Statisties. 

Scientific Peer 
Review 

Between June 2008 and June 2009, the Standards and Review Offiee (SRO) handled the 
proeessing and seientifie peer review of the Institute’s fiseal year (EY) 2009 researeh 
aetivities, researeh and development eenter, and researeh training program grant 
eompetitions. Aeross those eompetitions, 917 applieations were seientifieally reviewed 
by 19 review panels eomprising 350 external reviewers. In addition, the SRO managed 
the external seientifie peer review of 38 applieations for the Institute’s EY 2009 
Statewide Longitudinal Data System grant eompetition. 



Fourth Annuai 
lES Research 

The Institute held its fourth edueation researeh eonferenee June 7-9, 2009, in 
Washington, DC. More than 1,000 grant and eontraet awardees, polieymakers, leaders 
in the field of edueation researeh, and lES staff attended. Conferenee partieipants had 
the opportunity to hear about the new direetion of edueation at the federal level from 
Seeretary of Edueation Arne Dunean and Ceeilia E. Rouse, Member of the President’s 
Couneil of Eeonomie Advisers. Attendees also heard thoughtful opening remarks from 
National Board for Edueation Seienees viee ehair Jonathan Baron and the inaugural 
pubUe address of newly eonfirmed lES Direetor John Q. Easton. The eonferenee 
featured 21 panel and open-forum sessions and nearly 500 poster sessions presenting 
lES-supported researeh. This year’s researeh eonferenee represented a milestone in 
the Institute’s effort to build a national eommunity of edueation researehers, with 
eonferenee attendanee having grown from approximately 500 in 2006 to more than 
1,000 in 2009. 
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INSTITUTE OF EDUCATION SCIENCES CENTERS 

National Center tor 

Education Statistics 
(NCES^ 

NCES is the primary federal entity for eoUeeting, analyzing, and disseminating statistieal 
data related to edueation in the United States and in other nations. NCES eonsists of 
four divisions: Early Childhood, International, and Crosseutting Studies; Assessment; 
Elementary/Seeondary and Libraries Studies; and Postseeondary Studies. 

PROGRAM HIGHLIGHTS 

Longitudinal Studies NCES has initiated its next installment of the assessment of 
early ehhdhood development and learning, the Early Childhood Longitudinal Study of 
the Kindergarten Class of 2010-2011 (ECLS-K:2011). The ECLS-K:2011 is designed to 
provide eomprehensive and reliable data that ean be used to deseribe and to better 
understand ehildren’s development and experienees in the elementary grades, and how 
early experienees relate to later development, learning, and experienees in sehool. It 
will go beyond NCES’s previous early ehildhood longitudinal studies by eoUeeting data 
annuaUy from kindergarten through fifth grade. One of the major drawbaeks to analyzing 
ehUd development and performanee gains in the previous kindergarten studies was 
the laek of information on some key grades (e.g., seeond and fourth). The first data 
eoUeetion is seheduled for faU 2010 with a nationaUy representative sample of 21,000 
entering kindergartners in 900 sehools. 

Statewide Longitudinal Data System Grant Program (SLDS) The SLDS is designed 
to aid state edueation ageneies in developing and implementing longitudinal data 
systems to enhanee the ability of states to effieiently and aeeurately manage, analyze, 
and use edueation data, ineluding individual student reeords. These data systems wiU 
help states, distriets, sehools, and teaehers make data-driven deeisions to improve 
student learning, as weU as faeilitate researeh to inerease student aehievement and elose 
aehievement gaps. AU 50 states, five territories, and the Distriet of Columbia are eUgible 
to apply for the eooperative agreement grants. Previously, awards totaling $118.8 mUfion 
went to 27 states for the design, development, and implementation of K-12 statewide 
edueation data systems. New awards were made in 2009 to 27 states, ineluding 15 that 
had not previously been grant reeipients, bringing the total amount awarded to $265.2 
mUlion. Key programmatie features introdueed in the new awards inelude inelusion 
of pre-K through postseeondary edueation, system interoperability aeross states, and 
student-teaeher reeord linkages. The Ameriean Reeovery and Reinvestment Aet of 
2009 (ARRA) reinforees the eommitment to SLDS as an important tool for edueation 
improvement by authorizing an additional $245 million for a fourth grant eompetition. 
NCES provides guidanee and tec hn ical assistance to all of these state efforts. 
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National Assessment of Educational Progress (NAEP) NCES prepared Achievement 
Gaps: How Black and White Students in Public Schools Perform in Mathematics and 
Reading on the National Assessment of Educational Progress, a report detailing the 
size of the aehievement gaps between Blaek and White students at both the national 
and state levels, and how those aehievement gaps have changed over time. This is the 
first NAEP report focusing exclusively on Black and White achievement gaps, and the 
first containing state-level trends in these gaps. The analysis focuses on public school 
students at the fourth- and eighth-grade levels in both reading and mathematics, using 
NAEP scores as a common yardstick. State gap trends are presented since 1992 for 
reading, and since 1990 for mathematics. 



National Center tor 
Education Research 
fNCER^ 

NCER supports rigorous research that contributes to the solution of significant 
education problems in the United States. Through its research initiatives and the 
National Research and Development Centers (R&D Centers), NCER engages in 
research activities that result in the provision of high-quality education for aU children, 
improvement in student academic achievement, reduction in the achievement gap 
between high-performing and low-performing students, and increased access to and 
opportunity for postsecondary education. NCER research examines the effectiveness of 
education programs, practices, and policies, including the application of technology to 
instruction and assessment. The goal of NCER research programs is to provide scientific 
evidence of what works, for whom, and under what conditions. 

PROGRAM HIGHLIGHTS 

New Awards Eiscal year (EY) 2009 was a banner year for NCER. The Center hit 
a milestone, awarding 100 new grants across its research and research training 
competitions, with an overall funding rate of 16 percent. By comparison, NCER awarded 
only about one-fourth as many grants in EY 2002, when the overall funding rate was 9 
percent. 

Evaluation of State and Local Education Programs and Policies NCER recognizes 
that evidence-based choices for aU of the daily decisions that education decisionmakers 
and practitioners must make do not yet exist. Eurthermore, education leaders cannot 
always wait for scientists to provide answers. One solution to this dilemma is for the 
education system to integrate rigorous evaluation into the core of its activities. However, 
states and districts generally do not have the resources to conduct such activities. In 
EY 2009, NCER held the first competition for its research program on the Evaluation 
of State and Local Education Programs and Policies. Eor this program, NCER expects 
state or local education agencies to partner with researchers to evaluate the effects of 
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a program that the state or distriet plans to implement, or a program that has already 
been implemented but is not yet universal. Five projeets were awarded this year to 
evaluate programs that state or loeal edueation leaders are implementing and are 
invested in learning if they work. Two of the projeets will evaluate the effeets of state 
prekindergarten programs — one in Tennessee and the other in New Jersey. The foeus of 
a third projeet is on Indiana’s interim diagnostie assessment system — a system through 
whieh kindergarten through eighth-grade students take three formative assessments 
eaeh year. The evaluation will determine whether using the diagnostie assessment 
system improves student aehievement in reading, mathematies, seienee, and soeial 
studies. The fourth projeet involves eharter sehools in Colorado that use the Core 
Knowledge eurrieulum; its purpose is to evaluate the impaet on aehievement in the 
primary grades. The fifth projeet will evaluate the effeets of ninth-grade aeademies 
on aeademie performanee and engagement throughout high sehools in the Broward 
County Sehool Distriet in Florida. 

Postsecondary Education Research Since 2007, NCER has supported research that 
focuses on increasing at-risk students’ access to, persistence in, and completion of 
postsecondary education. One l in e of research addresses strategies for improving high 
school students’ preparation for and interest in postsecondary education. For example, 
NCER-funded researchers are evaluating mentoring and counseling programs in 
California and Vermont that are intended to motivate students who otherwise would not 
be considering college to apply; both programs provide assistance with the application 
process. Researchers in the National Center for Postsecondary Research (NCPR), one 
of the 13 national R&D Centers funded by NCER, are examining the effects of dual 
enrollment programs (i.e., simultaneous enrollment in high school and college courses) 
on student outcomes in Elorida. Another line of research concerns programs intended 
to help students who have graduated from high school but are not well prepared for 
postsecondary work. In fall 2000, about 28 percent of freshmen entering college, and 
42 percent of those entering community college, took a remedial course. Last year, both 
NCPR and other NCER-funded researchers reported that current remediation courses 
do not contribute to students’ completion of a degree. Currently, a team of NCPR 
researchers is working with the Texas Higher Education Coordinating Board to evaluate 
a Texas summer bridge program that is intended to provide intensive developmental 
education services to students with weak academic skills during the summer before 
they begin college. NCER’s postsecondary education research portfolio is growing and 
will yield findings that address important policy-relevant issues. 
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National Center tor 
Special Education 
Research rNCSER^ 

In December 2004, Congress reauthorized the Individuals with Disabilities Education Act 
(IDEA) and, in doing so, authorized NCSER. NCSER sponsors a comprehensive program 
of special education research designed to expand knowledge and understanding about 
infants, toddlers, and children with disabilities. NCSER is also charged with improving 
services provided under IDEA and with evaluating IDEA’S implementation and 
effectiveness. 

PROGRAM HIGHLIGHTS 

Advancing Science in Special Education Since its first research grant competition in 
EY 2006, NCSER has awarded more than 130 research grants. The funded projects cover 
a broad range of issues related to educating students with disabilities, including visual 
and hearing impairments, autism spectrum disorder, behavioral disorders, and learning 
disabilities. NCSER is beginning to build substantial research portfolios to address the 
needs of young children with disabilities and the needs of children with emotional and 
behavioral disorders. 

NCSER has invested more than $60 mi ll ion in research to improve outcomes for infants, 
toddlers, and young children with (or at risk for developing) disabilities. With more 
than 30 individual grants and one R&D Center, these projects cover a wide range 
of disabilities, including low-incidence disabilities such as deafness as weU as high- 
incidence disabilities such as language impairments. As the projects mature, NCSER 
will make a strong contribution to advancing knowledge in early intervention and early 
childhood special education. 

With 25 grants totaling more than $52 million, NCSER has made a considerable 
investment in improving social and behavioral outcomes for students with (or at risk for 
developing) disabilities. NCSER researchers have found that early intervention programs 
can improve students’ classroom behavior, social skills, and academic engaged time. 
NCSER is also advancing the behavioral assessment field through grants that support 
the development of measures for screening and progress monitoring for behavioral and 
emotional problems. 

Research and Development Centers NCSER’s first two special education R&D 
Centers began operation this year. The National R&D Center on Serious Behavior 
Disorders at the Secondary Level is developing and evaluating a package of 
interventions designed to reduce the significant behavioral and academic challenges 
experienced by high school students with serious emotional and behavior disorders. 

The Center for Response to Intervention in Early Childhood is conducting a focused 
research program to develop and evaluate intensive interventions for preschool 
language and early literacy skills, and to develop and validate an assessment system 
li nked to these interventions. 
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In March 2009, NCSER announced a competition for two additional R&D Centers. 

The first center wfil conduct a research program that identifies the academic growth 
trajectories of students with disabilities. The research program wiU also develop and 
test methods for use in accountability systems to practically and accurately measure 
academic growth for students with disabilities. The focus of the second center is on 
improving instruction for students with difficulties in mathematics. This center wiU 
examine underlying cognitive processes that impede performance in students with 
mathematics difficulties for the purpose of identifying possible targets for intervention. 
It will also develop and test innovative instructional approaches or other interventions 
for students with mathematics difficulties, utilizing cognitive research on mathematics 
disabilities and mathematics processing. 



National Center tor 
Education Evaluation 
and Regional 
Assistance 

Tlie mission of NCEE under the Education Sciences Reform Act of 2002 (ESRA) is to 

• conduct evaluations of federal education programs administered by the 
Secretary; 

• provide research-based technical assistance to educators and policymakers; 

• support synthesis and wide dissemination of the results of evaluation and 
research; and 

• encourage the use of scientifically valid education research and evaluation 
throughout the United States. 

ESRA further requires that NCEE evaluations “adhere to the highest possible standards 
of quality for conducting scientifically valid education evaluation,” where “scientifically 
valid” is used to describe evaluations that “employ experimental designs using random 
assignment, when feasible, and other research methodologies that allow for the 
strongest possible causal inferences when random assignment is not feasible,” and that 
there be rigorous peer review. In accordance with this requirement, aU reports from 
NCEE evaluations are peer reviewed by the lES Standards and Review Office. ESRA 
also required that the Commissioner of NCEE establish an external peer review for all 
Regional Educational Laboratory (REE) program research reports to ensure that they 
meet lES standards for research and evaluation before being released. Accordingly, study 
reports from REE randomized controlled trials (RCTs) are required to pass the NCEE 
external peer review process and also undergo review from the lES Standards and 
Review Office. 
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PROGRAM fflGHUGHTS 

NCEE’s Evaluation Studies Division The National Board for Education Sciences: 
5-Year Report, 2003 Through 2008 noted that NCEE had 26 large evaluations 
eompleted or under way as of Oetober 2008, and that 22 of the evaluations 
used randomized eontrolled designs to test eomparisons between the treatment 
intervention(s) and the eontrol eondition. 

In noting that only one randomized field trial was used in 16 U.S. Department of 
Edueation (ED) evaluations in EY 2000, and that during the period from 1995 through 
1997, only 5 of 51 evaluations of federal programs used the randomized eontrolled 
design, the report said, “The Institute should be applauded for its efforts to greatly 
improve the rigor of impaet evaluations of ED programs.”* 

Sinee the beginning of EY 2009 (Oetober 2008), NCEE has released nine impaet 
evaluation reports: 

• Effectiveness of Selected Supplemental Reading Comprehension Interventions: 
Impacts on a Eirst Cohort of Eifth-Grade Students (May 2009); 

• Evaluation of the DC Opportunity Scholarship Program: Impacts After Three 
Years (Mareh 2009); 

• Impact Evaluation of the U.S. Department of Education’s Student Mentoring 
Program (Mareh 2009); 

• Achievement Effects ofEour Early Elementary School Math Curricula: 
Eludings from Eirst-Graders in 39 Schools (Eebruary 2009); 

• Effectiveness of Reading and Mathematics Software Products: Eindings Erom 
Two Student Cohorts (Eebruary 2009); 

• An Evaluation of Teachers Trained Through Different Routes to Certification 
(Eebruary 2009); 

• Enhanced Reading Opportunities: Eindings Erom the Second Year of 
Implementation (November 2008); 

• Reading Eirst Impact Study Einal Report (November 2008); and 

• Impacts of Comprehensive Teacher Induction: Results Erom the Eirst Year of a 
Randomized Controlled Study (Oetober 2008). 

The new eongressionally mandated evaluation of the Teaeher Ineentive Eund program 
eontained in ARRA is the latest addition to the NCEE evaluation portfolio. 



* National Board for Education Sciences. (2008). National Board for Education Sciences: 5-Year Report, 2003 Through 
2008 (NBES 2009-601 1). National Board for Education Sciences, Institute of Education Sciences, U.S. Department of 
Education. Washington, DC, p. 11. 
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To make the findings from these evaluation reports aeeessible to a broader audienee, 
NCEE is produeing “NCEE Evaluation Briefs” for distribution at NCEE eonferenees. Eight 
are eurrently available under the “Resourees” seetion of the NCEE website at 
http : //ies . ed . gov/neee/pubs . 

NCEE’s Knowledge Utilization Division — ^Dissemination, Synthesis, and 
Advancement of the Use of Scientiftcally Valid Education Research and What 
Works Clearinghouse (WWC) Evaluation The WWC has already reeeived substantial 
attention in the report prepared by the blue-ribbon panel of leading experts in rigorous, 
partieularly randomized, evaluations eommissioned by the NBES. The NBES eharge to 
this Expert Panel was to eonduet a foeused study addressing the fundamental question 
of whether the WWC ’s evidenee review proeess and reports are seientifieally valid and 
provide aeeurate information about the strength of evidenee of meaningful effeets on 
important edueational outeomes. The Expert Panel’s report was released in Oetober 
2008 and was diseussed at the January 2009 NBES Board meeting. Sinee then, WWC 
has eontinued to expand both the range and volume of its publieations to reaeh more 
audienees interested in what works evidenee. 

In Mareh 2009, noted edueation reporter Debra Viadero wrote about the turnaround of 
the WWC in her report for Education Week’s Inside Sehool Researeh blog. In an entry 
titled “Is ‘What Works’ Living Up to Its Name?”, Viadero wrote this: 

We’re going to have to stop eaUing the U.S. Department of Edueation ’s What 
Works Clearinghouse the “nothing works” elearinghouse. Set up in 2002 to vet 
researeh on edueational programs and praetiees, the elearinghouse got that 
unfortunate nic kn ame because so few of its early reviews turned up educational 
interventions that were any more effective than what educators were already 
doing. 

This new statistic from Mathematica Policy Research Inc., the Princeton, NJ, com- 
pany that operates the clearinghouse, suggests that times have changed: Of the 
100-plus reports now posted on the clearinghouse Web site, 62 percent have at 
least one outcome that’s positive. 

If you go to the Web site to see for yourself, check out the nifty new search tools . 
They can spit out charts showing you how all the interventions in a particular 
topic area — say, reading or dropout prevention — stack up against one another by 
Wliat Works standards. You can also customize the results by grade level, student 
population, or the learning outcome that interests you. 

With aU those new beUs and whistles, the clearinghouse ought to come up with 
a slogan — perhaps “put what works to work for you” — to bury the “nothing 
works” moniker once and for all. 
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The WWC also creates Practice Guides that offer practical recommendations from 
expert panelists who are chosen because of their significant experience as researchers 
or practitioners in a particular area of education. These experts apply WWC review 
processes and standards as they cull the literature, decide on recommendations, and 
determine the level of evidence in support of their recommendations. These guides are 
especially popular and have become more so due to the dissemination events organized 
by the RELs. Plans are under way to extend the Practice Guides by providing WWC 
Practice Briefs that offer expanded examples of how to put recommendations into 
practice. 

WWC Quick Reviews, an entirely new WWC product, were initiated in late 2008 to 
inform educators, policymakers, researchers — in fact, all who follow education news — 
about what studies reported in the national media claim and whether the studies, based 
on publicly available documents, appear to meet WWC evidence standards. These 
studies are often about interventions that may not fit into any existing WWC review area. 

Intervention Reports are now produced on interventions that may have name 
recognition but could not be reported on by the WWC because they lacked supporting 
WWCTevel evidence. These brief reports explain what studies were reviewed and why 
the studies did not meet WWC standards. WWC continues to produce Intervention 
Reports in the topic areas of early childhood education, beginning reading, middle 
school math, dropout prevention, and English language learners. WWC is launching 
seven new topic areas in adolescent literacy, children with emotional or behavioral 
disabilities, early childhood education interventions for children with disabilities, 
high school math, out-of-school time, students with learning disabilities, and teacher 
professional development. 

Finally, WWC, now established as the systematic review organization that covers the 
education sector, is making new outreach linkages. WWC is the primary producer of 
lES reports of great interest to practitioners served by the R ET, program, and is very 
active in providing speakers for special RET, dissemination events. WWC has established 
relationships with other leading experts in the field of systematic reviews. WWC is 
planning to increase its dissemination activities by hosting events, such as forums and 
webinars, and developing other events with its dissemination partners. 

Regional Educational Laboratory (REL) Program Since 2005, the REE program has 
been refocused to meet the scientific requirements of ESRA, particularly the 

• mandatory requirements for peer review of all research plans and products; 

• increased transparency in monitoring regional needs and the responses by the 
laboratories; and 
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• provision of technical assistance to REL customers through published reports 
from short-term “fast-response” projects and through the conduct of large-scale 
rigorous studies. 

A key issue for those participating in the RELs was the new NCEE peer review process, 
which applied lES scientific evidence standards to the issues confronting local and state 
education entities. Eor example, typical questions include 

• What dropout prevention programs have been tried in our region and which 
ones are effective? 

• Can we develop a list of teacher preparation programs that provide quality 
instruction in special education? 

Such questions, to be answered without relying on personal opinions or superficial 
assessments, require a methodologically rigorous review of available research. To 
advance the central distinction between methods that adequately control for factors 
(some observable, many not observable) that are “causing” a particular outcome and 
methods that pay little attention to these factors, the laboratories were asked to carry 
out rigorous studies of new practices, curriculums, and teacher quality enhancement 
strategies in which state and local education leaders expressed strong interest. Each of 
the 10 laboratories typically launched 2 to 3 such 5-year studies, for a total of 25 RCTs 
in all. Many of the studies focused on mathematics, science, or literacy; some focused on 
early childhood and high school interventions. 

A real benefit in having the RELs conduct rigorous studies is that it demonstrates that 
the laboratories can offer new evaluation resources and skfils that are focused precisely 
on what can be learned from studies within one’s own state or neighboring jurisdictions 
at a time when school resources are tight. 

The RET, evaluations adhere to a common set of reporting standards that clearly 
separates the confirmatory findings from exploratory analyses that do not allow 
causal inference. This distinction will help others produce more reliable reports, 
communicating clearly the effects, versus the suggested associations, of an intervention. 
Peer reviewed reports from these studies are expected in 2009-2011. 

Widespread dissemination of scientific research findings is integral to the mission of 
the REL program. To better reach the laboratories’ constituency of practitioners, a new 
dissemination system was established by holding REL forums. These forums provide 
practitioners with direct access to findings from lES-funded research and evaluation. 
Speakers at the events are experts who have authored lES publications, including the 
highly popular lES Practice Guides produced by the WWC. 

• As of June 2009, 26 REL forums have been held throughout the country, with 
many more events scheduled within the next 6 months. These forums provide a 
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rich opportunity not only to stimulate discussion regarding the implications or 
pertinence of scientifically rigorous evidence but also to try out new approaches 
to bridging the gap between practitioners’ terms of reference and research and 
evaluation experts’ terms and frameworks. 

• As of June 2009, 82 peer reviewed Issues & Answers reports (results of short- 
term fast-response projects) have been released, covering a wide range of critical 
issues including, value-added modeling, trend analyses, the alignment of state 
and NAEP standards, teacher preparation programs, English language learners, 
students with disabilities, and many other priorities. 

Education Resources Information Center (ERIC) ERIC, the other knowledge 
utilization project dating from the 1960s, went through a highly significant and cost- 
effective change in 2003 to become a modern electronic library. Greater scale and 
efficiencies were made possible by closing 17 ERIC clearinghouses and replacing them 
with a single ERIC literature collection point. The overhaul allowed for expansion 
of the collection to include more journals (using selection criteria established by 
expert opinion) and rapid electronic archiving of new resources. The new ERIC uses 
microfiche digitization to feature archived ERIC materials from the 1966-1992 historic 
ERIC collection. ERIC, which can be accessed at http : //www. eric . ed . gov , was recently 
commended as “a Best Eree Reference Web Site” by the Reference and User Services 
Association of the American Library Association, which cited ERIC’s powerful search 
tools and other features. 

NCEE Methods Working Group — Advancing the Apphcation of Evaluation 
Methods and EstahUshing Scientifically Rigorous Standards in the Eield In 2006, 
the Commissioner established the NCEE Methods Working Group to work on issues and 
strategies to assure that NCEE evaluations take advantage of the latest developments 
in evaluation methods. The group engages in discussion on newly emerging issues 
and identifies ways to conduct methodological investigations to resolve or adapt 
new techniques. Some of the issues brought to the Methods Working Group grew 
out of WWC deliberations and by the R E T program’s RCTs. The RE T RCTs essentially 
doubled the number of experimental studies being conducted under NCEE. The R ET 
RCT initiative brought education researchers into close working relationships with 
experimental research experts. The large number of experimental NCEE evaluations 
also brought demands for consistent guidance and reporting requirements across the 
various entities responsible for the first field trials on many issues. The results of the 
investigations are published as commissioned, peer reviewed papers under the series 
title Technical Methods Reports. These reports are posted on the NCEE website at 
http : //ies . ed.gov/ncee/pubs . They are specifically designed for use by researchers, 
methodologists, and evaluation specialists. To date, four reports have been released, and 
nine others are expected to be considered for publication in 2009-2011. 
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Midstream Assessment of the lES/NCEE Evaluation Program This assessment took 
stock of the first cohort of federal evaluation studies to follow lES scientific standards. 
Preliminary findings from the assessment were presented at the September 2008 NBES 
meeting. The assessment considered 24 evaluation studies of federal programs under 
NCEE. The main recommendations included in the assessment report are as follows: 

• NCEE should sustain the production of scientifically rigorous evidence to help 
others judge the effectiveness of federal program investments. 

• Congress and others will obtain important indicators of new program initiatives’ 
potential cost-effectiveness if NCEE is asked to conduct pilot tests of the new 
initiatives prior to full-scale implementation, or “evidence-guided innovation .” 

• Dissemination of research findings has received attention through the NC EE 
establishment of evaluation reporting guidelines and the WWC additions, such as 
Quick Reviews. 

• NCEE evaluations of teacher quality strategies urge more systematic work on 
measuring teacher quality in general. 

• Infrastructure issues to improve the usefulness of NCEE evaluations are detailed 
below. 

o Continue efforts to make major advancements in the development and 
dissemination of standards of evidence. 

o Give attention to improving the consistency of outcome measures, 
especially across state tests. 

o Limit the use of effect size measures for establishing the rationale and 
design base for education evaluations and move to using metrics more 
directly tied to policy contexts (e.g., changes in learning growth). 

o Add estimates of resource use/costs to evaluation studies. 

o Develop and test simulation models to allow school entities to estimate 
impacts likely in their school contexts. 

o Avoid defaulting to using the school as the unit of assignment, or 

assuming that control group contamination is sufficiently large to avoid 
using classrooms, teachers, or students as the unit of assignment. 

o Evaluate the cost-effectiveness of conducting extensive mediator 
analyses and implementation/process analyses. 

It should be noted that the three authors of the assessment are also drafting extended 
memorandums on selected topics. These memorandums will more fully inform 
the research and evaluation communities about the recommendations made in the 
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assessment report, the rationale for eaeh reeommendation, and speeifie examples. 
Ineluded will be reeommendations to introduee more poliey-relevant outeome 
measures, to develop simulation models of likely effeets in varying sehool eontexts, 
and to propose an experiment to test teaeher praetiees for their impaet on student 
aehievement. 



BOARD ACTIVITIES 

What Works 
Clearinghouse 
(WWC) Evaluation, 

In July 2008, the National Board for Edueation Seienees (NBES) eonvened an expert 
panel to eonduet a foeused study addressing the fundamental question of whether 
the WWC’s evidenee review proeess and reports are seientifieally valid. That is, do 
they provide aeeurate information about meaningful effeets on important edueation 
outeomes? 

Based on analysis and investigation, the panel generally eoneluded that 

1. WWC proeedures and proeesses for identifying and extraeting information 
from intervention studies are generally well doeumented and foUow reasonable 
standards and praetiees for systematie reviews; and 

2. WWC Intervention and Topie Area Reports provide sueeinet and meaningful 
summaries of the evidenee on the effeetiveness of speeifie edueation 
interventions. 

The panel also provided the following reeommendations: 

1 . Full Review. The panel reeommends that the Institute of Edueation Seienees 
GES) eommission a full review of the WWC, ineluding a review of the WWC’s 
mission and of the WWC Praetiee Guides that the panel did not evaluate. The 
panel also reeommends that lES eonsider instituting a regular review proeess to 
ensure that WWC is using the most appropriate standards in its work. 

2. Protocol Templates. The panel reeommends that the WWC review and update 
the protoeol templates, foeusing on the following issues: 

(i) standards for erossover and assignment noneomplianee, and for adjusting 
intention to treat effeets aeross studies. 

(ii) 
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standards for doeumenting the program reeeived in the eontrol arm 
of randomized-eontrol trials (or by members of the eomparison 
group in quasi-experimental designs), and potentially ineorporating 
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this information when making comparisons across studies and/or 
interventions. 

(iii) revised standards for multiple comparisons. We recommend that the 
WWC review the treatment of multiple comparisons in light of a recent 
research report by Peter Schochet titled Guidelines for Multiple Testing 
in Impact Evaluations. 

(iv) attrition standards. We recommend that the WWC reconsider the current 
process of setting different attrition standards in different topic areas. 

(v) potential conflicts of interest. We recommend that the WWC establish 
a new protocol to keep track of potential conflicts of interest, such as 
cases where a study is funded or conducted by a program developer, and 
consider making that information available in its reports. 

(vi) randomization. We recommend that the WWC precisely define the 
standards for “randomization” in a multilevel setting. 

3. Documentation of Search Process. The panel recommends that the WWC 
expand the protocol templates to specify more explicit documentation of the 
actual search process used in each topic area, and to maintain a record of the 
results of the search process that can be used to guide decisionmaking on future 
modifications of the search process. 

4. Reliability of Eligibility Screening. The panel recommends that the WWC 
conduct regular studies of the reliability of the eligibility screening process, 
using two independent screeners, and use the results from these studies to 
refine the eligibility screening rules and screening practices. 

5. Documentation of Screening Process. The panel recommends that WWC 
reports include a QUOROM-type flow chart documenting the flow of studies 
through each review and the number of studies excluded at each point, and a 
Table of Excluded Studies that lists specific reasons for exclusion for each study. 

6. Misalignment Adjustment The panel recommends that in cases where a study 
analysis is “misaligned,” WWC staff request that study authors re-analyze their 
data correctly, taking into account the unit of randomization and clustering. We 
recommend that the results from the process be compared to the simple ex post 
adjustment procedure currently specified to develop evidence on the validity of 
the latter. 

7. Combining Evidence Across Multiple Studies. We recommend that the WWC 
re-evaluate procedures for combining evidence across studies, with specific 
attention paid to the issue of how the rules for combining evidence can be 
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optimally tuned, given the objeetives of the WWC review proeess and the 
sample sizes in typieal studies for a topie area. 

8. Reporting. 

(i) The panel reeommends that published reports on the website inelude 
the topie area protoeols, as well as more information on the sereening 
proeess results that led to the set of eligible studies aetuaUy summarized 
in the Topie Area Reports. 

(ii) The panel reeommends that the WWC make readily available its 
Procedures and Standards Handbook, ineluding appendixes, as weU as 
all other relevant doeuments that establish and doeument its poheies and 
proeedures. 

9. Practice Guides. The panel reeommends that the Praetiee Guides — whieh 
eontain material that does not meet the high standards of evidenee for other 
WWC produets — be elearly separated from the Topie Area and Intervention 
Reports. 

10. Outreach and Collaboration With Other Organizations. The panel 
reeommends that the WWC build and maintain a relationship with national and 
international organizations foeusing on systematie reviews, speeifieaUy with the 
goals of having Review Team leaders engage in the broader seientifie eommunity 
and bringing the latest standards and praetiees to the WWC. The panel also 
reeommends that the WWC eonvene working groups with a mixture of 
researehers, ineluding speeiaUsts in edueation researeh and systematie reviews, 
to address the development of new standards for the review and synthesis of 
studies. 

The WWC evaluation report is available online at http : //ie s . ed . gov/direc tor/board/pdf/ 
panelreport . pdf 
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National Board for 
Education Sciences 
5-Year Report 

Tlie Education Sciences Reform Act of 2002 (ESRA) stipulated that, 5 years after 
enactment, the Board was to submit to the Director, the Secretary, and the appropriate 
congressional committees a report that included any recommendations regarding 
actions that may be taken to enhance the ability of lES to carry out its priorities and 
mission. In November 2008, a 5-Year Report was published with the Board’s evaluation 
of the Institute. The Board examined the ways in which and the extent to which the 
Institute has been successful in advancing the rigor of education research, improving its 
relevance, and facilitating evidence-based decisionmaking. 

On the basis of its evaluation, the members of NBES concluded that within a relatively 
brief period, lES has made exceptional or substantial progress in improving the rigor and 
relevance of education research and the dissemination of practical information. 

ADVANCING THE RIGOR OF EDUCATION RESEARCH 

Die Board’s determination of “exceptional progress” in advancing the rigor of 
education research was based on the following evidence: 

1 . the high standards reflected in the peer review system; 

2. the strong external ratings on the quality of the funded research grants; and 

3. the high quality of the research designs of the evaluations contracted through 
the Institute’s National Center for Education Evaluation and Regional Assistance 
(NCEE). 

IMPROVING THE RELEVANCE OF EDUCATION RESEARCH 

The Board’s determination of “substantial progress” in improving the relevance of 
education research was based on the following: 

1 . satisfactory ratings of the relevance of funded research projects by education 
leaders and administrators; 

2. establishment of long-term focused programs of research that address 
fundamental education issues in our nation (e.g., improving reading, writing, 
mathematics, and science achievement); 

3. the work of the National Research and Development Centers in key policy areas; 

4. dramatic improvements in the timeliness of the release of data from the 
Institute’s National Center for Education Statistics (NCES); 

5. liigh ratings of the relevance of NCES reports; and 
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6. increased efforts to improve the timeliness of the release of NCEE evaluation 
reports. 

FACILITATING EVIDENCE-BASED DECISIONMAKING 

The Board’s determination of “substantial progress” in facilitating evidence-based 
decisionmaking was based upon the quality and use of the systems and programs 
the Institute has created (e.g.,WWC, College Navigator) or revamped (e.g., Education 
Resources Information Center) to disseminate practical i nf ormation to education 
leaders, practitioners, parents, and students. 

Although the Board finds that the Institute has been instrumental in improving the rigor, 
relevance, and accessibility of federally funded education research, much remains to be 
done to institutionalize the gains made and to build on them. No matter how technically 
sound research activities might be, if they do not address the issues and questions that 
are of concern to education policymakers and practitioners, the research will not be 
used to in form education policy and practice. The main message from the evaluation 
is that the Institute’s leadership and staff have successfully executed the priorities and 
mission of lES and that, going forward, lES should maintain the direction that Congress 
has articulated. 

The Board’s 5-Year Report is available at http : //ies . ed. gov/director/board/ 
reports/2009601 1/index. asp . 

NEXT STEPS 

Over time, the Board has approved various resolutions (see appendix B). These 
resolutions serve as guidance for Congress, the Secretary, and the Director of IES. The 
Board will advise the Director of IES on recommendations for center commissioner 
appointments and for approving his research priorities, as required by law. 

In addition, the Board looks forward to a prompt reauthorization of ESRA. An annotated 
version of ESRA, with recommendations for changes, can be found in the Board’s 5-Year 
Report. Recommended changes include (a) clarifying ambiguous descriptions, (b) 
establishing processes to avoid long-term vacancies, (c) maintaining the peer review 
process under the Office of the Director, (d) estabUshing procedures to buffer the 
Institute further from outside influences and to maintain the integrity and independence 
of IES as a science agency, (e) changing the definition of “scientifically based research” 
to align with more recent language in other legislation, and (f) establishing the Institute 
as the lead agency for congressionaUy authorized scientific education research under 
various education laws. 
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CONCLUSION 

Over the past 7 years, a new direetion has been set for edueation researeh. It is 
important to retain the foeus of the Institute of Edueation Seienees GES) on funding 
researeh that meets high standards of seientifie rigor while addressing questions of 
signifieanee from the edueation field. 



CHAIR’S MESSAGE 

As is widely reeognized, lES made great strides during its first 6 years of operation. It signifieantly 
upgraded the quality and reliability of edueation researeh and evaluation. The Institute began a 
new phase on June 1, 2009, when John Q. Easton beeame its direetor. The NBES looks forward 
to working elosely with Dr. Easton so that the path of federal edueation researeh ean eontinue its 
upward momentum. NBES has been handieapped by not being at full strength, which we hope 
will be corrected soon. 

— Eric A. Hanushek 
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APPENDIX A 

Members of the 
National Board for 
Education Sciences 
(as of Juiy 1 , 2009^ 

Mr. Jonathan Baron 

Coalition for Evidence-Based Policy 
Washington, DC 

Dr. Carol A. D’Amico 

Conexus Indiana 
IndianapoUs, IN 

Dr. David C. Geary 

University of Missouri 
Columbia, MO 

Mr. F. Philip Handy 

Strategic Industries, EE C 
Winter Park, EE 

Dr. Eric A. Hanushek 

Hoover Institution 
Stanford University 
Stanford, CA 

Dr. Sally E. Shaywitz 

Department of Pediatrics 

Yale University School of Medicine 

New Haven, CT 

Ex Officio 

Director of the Institute of Education Sciences 

Each of the Commissioners of the lES National Education Centers 

Director of the National Institute of Child Health and Human Development 

Director of the Census Bureau 

Director of the National Science foundation 

Commissioner of the Bureau of Eabor Statistics 
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APPENDIX B 

Approved National 
Board for Education 
Sciences 

1 . Congress, in authorizing and funding evaluations of federal edueation programs, 
should require [program] grantees, as a eondition of grant award, to partieipate 
in the evaluation if asked, ineluding the random assignment to intervention and 
eontrol groups as appropriate. (April 2005) 

2. Congress and the U.S. Department of Edueation should ensure that individual 
student data ean be used by researehers (with appropriate safeguards for 
eonfidentiality) in order to provide evaluations and analyses to improve our 
sehools. (September 2006) 

3- Congress should designate the Institute of Edueation Seienees, in statute, as the 
lead ageney for all eongressionaUy authorized evaluations of U.S. Department 
of Edueation programs, responsible for all operations, eontraets, and reports 
assoeiated with sueh evaluations. (September 2006) 

4. Congress should allow the U.S. Department of Edueation to pool funds 
generated by the 0.5 pereent evaluation set-aside from smaller programs. 
(September 2006) 

5. The U.S. Department of Edueation should use its “waiver” authority to build 
seientifieaUy valid knowledge about what works in K-12 edueation. (September 
2006) 

6. Congress should ereate, in statute, effeetive ineentives for federal edueation 
program grantees to adopt praetiees or strategies meeting the highest standard 
of evidenee of sizeable, sustained effeets on important edueational outeomes. 
(May 2007) 

7. Congress should revise the statutory definition of “seientifieaUy based researeh” 
so that it ineludes studies likely to produee valid eonelusions about a program’s 
effeetiveness, and exeludes studies that often produee erroneous eonelusions. 
(Oetober 2007) 

8. The Board wiU review and advise the lES Direetor on grant awards where the 
proposed grantee is seleeted out of rank order of applieant seores that result 
from peer review for seientifie merit. (January 2008) 

9. The Board eommends the Seeretary and the U.S. Department of Edueation for 
moving forward in developing new regulations and guidanee about how to 
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maintain confidentiality of educational data under the Family Educational 
Rights and Privacy Act (FERPA) while also providing for research uses of 
student and school data. The Department should finalize these regulations 
quickly, incorporating the major clarifications that have been submitted in 
comments. (May 2008) 

10. Congress should expand on the program of supporting statewide 

longitudinal data systems by requiring that states accepting funding under 
this program agree to make data in these systems available to qualified 
researchers (subject to FERPA) for the purpose of research that is intended 
to help improve student achievement. (May 2008) 
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